{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 探索性数据分析（EDA）\n",
    "\n",
    "\n",
    "有趣的数据集，包含球员和裁判之间的故事！\n",
    "\n",
    "\n",
    "数据集介绍点击： [here](https://osf.io/47tnc/).\n",
    "\n",
    "<img src=\"figures/f1.png\" alt=\"FAO\" width=\"290\" >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 任务：\n",
    "\n",
    "探索性数据分析（EDA）. 挑战目标: **这些裁判在给红牌的时候咋想的呢，会不会被跟球员的肤色有关?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format='retina'\n",
    "\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import matplotlib as mpl\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.pyplot import GridSpec\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os, sys\n",
    "from tqdm import tqdm\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "sns.set_context(\"poster\", font_scale=1.3)\n",
    "\n",
    "import missingno as msno\n",
    "import pandas_profiling\n",
    "\n",
    "from sklearn.datasets import make_blobs\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据简介：\n",
    "\n",
    "> 数据包含球员和裁判的信息，2012-2013年的比赛数据，总共设计球员2053名，裁判3147名，特征列表如下：\n",
    "\n",
    "> -- https://docs.google.com/document/d/1uCF5wmbcL90qvrk_J27fWAvDcDNrO9o_APkicwRkOKc/edit\n",
    "\n",
    "\n",
    "| Variable Name: | Variable Description: | \n",
    "| -- | -- | \n",
    "| playerShort | short player ID | \n",
    "| player | player name | \n",
    "| club | player club | \n",
    "| leagueCountry | country of player club (England, Germany, France, and Spain) | \n",
    "| height | player height (in cm) | \n",
    "| weight | player weight (in kg) | \n",
    "| position | player position | \n",
    "| games | number of games in the player-referee dyad | \n",
    "| goals | number of goals in the player-referee dyad | \n",
    "| yellowCards | number of yellow cards player received from the referee | \n",
    "| yellowReds | number of yellow-red cards player received from the referee | \n",
    "| redCards | number of red cards player received from the referee | \n",
    "| photoID | ID of player photo (if available) | \n",
    "| rater1 | skin rating of photo by rater 1 | \n",
    "| rater2 | skin rating of photo by rater 2 | \n",
    "| refNum | unique referee ID number (referee name removed for anonymizing purposes) | \n",
    "| refCountry | unique referee country ID number | \n",
    "| meanIAT | mean implicit bias score (using the race IAT) for referee country | \n",
    "| nIAT | sample size for race IAT in that particular country | \n",
    "| seIAT | standard error for mean estimate of race IAT   | \n",
    "| meanExp | mean explicit bias score (using a racial thermometer task) for referee country | \n",
    "| nExp | sample size for explicit bias in that particular country | \n",
    "| seExp |  standard error for mean estimate of explicit bias measure | \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Uncomment one of the following lines and run the cell:\n",
    "\n",
    "df = pd.read_csv(\"redcard.csv.gz\", compression='gzip')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(146028, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>playerShort</th>\n",
       "      <th>player</th>\n",
       "      <th>club</th>\n",
       "      <th>leagueCountry</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>position</th>\n",
       "      <th>games</th>\n",
       "      <th>victories</th>\n",
       "      <th>...</th>\n",
       "      <th>rater2</th>\n",
       "      <th>refNum</th>\n",
       "      <th>refCountry</th>\n",
       "      <th>Alpha_3</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>nIAT</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>nExp</th>\n",
       "      <th>seExp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>lucas-wilchez</td>\n",
       "      <td>Lucas Wilchez</td>\n",
       "      <td>Real Zaragoza</td>\n",
       "      <td>Spain</td>\n",
       "      <td>31.08.1983</td>\n",
       "      <td>177.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>Attacking Midfielder</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>GRC</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>712.0</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>750.0</td>\n",
       "      <td>0.002696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>john-utaka</td>\n",
       "      <td>John Utaka</td>\n",
       "      <td>Montpellier HSC</td>\n",
       "      <td>France</td>\n",
       "      <td>08.01.1982</td>\n",
       "      <td>179.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>Right Winger</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.75</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>ZMB</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>49.0</td>\n",
       "      <td>0.061504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>abdon-prats</td>\n",
       "      <td>Abdón Prats</td>\n",
       "      <td>RCD Mallorca</td>\n",
       "      <td>Spain</td>\n",
       "      <td>17.12.1992</td>\n",
       "      <td>181.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pablo-mari</td>\n",
       "      <td>Pablo Marí</td>\n",
       "      <td>RCD Mallorca</td>\n",
       "      <td>Spain</td>\n",
       "      <td>31.08.1993</td>\n",
       "      <td>191.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>Center Back</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ruben-pena</td>\n",
       "      <td>Rubén Peña</td>\n",
       "      <td>Real Valladolid</td>\n",
       "      <td>Spain</td>\n",
       "      <td>18.07.1991</td>\n",
       "      <td>172.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>Right Midfielder</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1897.0</td>\n",
       "      <td>0.001002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     playerShort         player             club leagueCountry    birthday  \\\n",
       "0  lucas-wilchez  Lucas Wilchez    Real Zaragoza         Spain  31.08.1983   \n",
       "1     john-utaka     John Utaka  Montpellier HSC        France  08.01.1982   \n",
       "2    abdon-prats    Abdón Prats     RCD Mallorca         Spain  17.12.1992   \n",
       "3     pablo-mari     Pablo Marí     RCD Mallorca         Spain  31.08.1993   \n",
       "4     ruben-pena     Rubén Peña  Real Valladolid         Spain  18.07.1991   \n",
       "\n",
       "   height  weight              position  games  victories    ...     rater2  \\\n",
       "0   177.0    72.0  Attacking Midfielder      1          0    ...       0.50   \n",
       "1   179.0    82.0          Right Winger      1          0    ...       0.75   \n",
       "2   181.0    79.0                   NaN      1          0    ...        NaN   \n",
       "3   191.0    87.0           Center Back      1          1    ...        NaN   \n",
       "4   172.0    70.0      Right Midfielder      1          1    ...        NaN   \n",
       "\n",
       "   refNum  refCountry  Alpha_3   meanIAT    nIAT     seIAT   meanExp    nExp  \\\n",
       "0       1           1      GRC  0.326391   712.0  0.000564  0.396000   750.0   \n",
       "1       2           2      ZMB  0.203375    40.0  0.010875 -0.204082    49.0   \n",
       "2       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "3       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "4       3           3      ESP  0.369894  1785.0  0.000229  0.588297  1897.0   \n",
       "\n",
       "      seExp  \n",
       "0  0.002696  \n",
       "1  0.061504  \n",
       "2  0.001002  \n",
       "3  0.001002  \n",
       "4  0.001002  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>height</th>\n",
       "      <td>145765.0</td>\n",
       "      <td>181.935938</td>\n",
       "      <td>6.738726</td>\n",
       "      <td>1.610000e+02</td>\n",
       "      <td>177.000000</td>\n",
       "      <td>182.000000</td>\n",
       "      <td>187.000000</td>\n",
       "      <td>2.030000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>143785.0</td>\n",
       "      <td>76.075662</td>\n",
       "      <td>7.140906</td>\n",
       "      <td>5.400000e+01</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>games</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>2.921166</td>\n",
       "      <td>3.413633</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.700000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>victories</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>1.278344</td>\n",
       "      <td>1.790725</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.900000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ties</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.708241</td>\n",
       "      <td>1.116793</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>defeats</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.934581</td>\n",
       "      <td>1.383059</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.800000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>goals</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.338058</td>\n",
       "      <td>0.906481</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.300000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yellowCards</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.385364</td>\n",
       "      <td>0.795333</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yellowReds</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.011381</td>\n",
       "      <td>0.107931</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>redCards</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>0.012559</td>\n",
       "      <td>0.112889</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rater1</th>\n",
       "      <td>124621.0</td>\n",
       "      <td>0.264255</td>\n",
       "      <td>0.295382</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rater2</th>\n",
       "      <td>124621.0</td>\n",
       "      <td>0.302862</td>\n",
       "      <td>0.293020</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>1534.827444</td>\n",
       "      <td>918.736625</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>641.000000</td>\n",
       "      <td>1604.000000</td>\n",
       "      <td>2345.000000</td>\n",
       "      <td>3.147000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <td>146028.0</td>\n",
       "      <td>29.642842</td>\n",
       "      <td>27.496189</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>1.610000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.346276</td>\n",
       "      <td>0.032246</td>\n",
       "      <td>-4.725423e-02</td>\n",
       "      <td>0.334684</td>\n",
       "      <td>0.336628</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>5.737933e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>19697.411216</td>\n",
       "      <td>127126.197143</td>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>1785.000000</td>\n",
       "      <td>2882.000000</td>\n",
       "      <td>7749.000000</td>\n",
       "      <td>1.975803e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seIAT</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.000631</td>\n",
       "      <td>0.004736</td>\n",
       "      <td>2.235373e-07</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.000151</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>2.862871e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>meanExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.452026</td>\n",
       "      <td>0.217469</td>\n",
       "      <td>-1.375000e+00</td>\n",
       "      <td>0.336101</td>\n",
       "      <td>0.356446</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>1.800000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>20440.233860</td>\n",
       "      <td>130615.745103</td>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>1897.000000</td>\n",
       "      <td>3011.000000</td>\n",
       "      <td>7974.000000</td>\n",
       "      <td>2.029548e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seExp</th>\n",
       "      <td>145865.0</td>\n",
       "      <td>0.002994</td>\n",
       "      <td>0.019723</td>\n",
       "      <td>1.043334e-06</td>\n",
       "      <td>0.000225</td>\n",
       "      <td>0.000586</td>\n",
       "      <td>0.001002</td>\n",
       "      <td>1.060660e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                count          mean            std           min          25%  \\\n",
       "height       145765.0    181.935938       6.738726  1.610000e+02   177.000000   \n",
       "weight       143785.0     76.075662       7.140906  5.400000e+01    71.000000   \n",
       "games        146028.0      2.921166       3.413633  1.000000e+00     1.000000   \n",
       "victories    146028.0      1.278344       1.790725  0.000000e+00     0.000000   \n",
       "ties         146028.0      0.708241       1.116793  0.000000e+00     0.000000   \n",
       "defeats      146028.0      0.934581       1.383059  0.000000e+00     0.000000   \n",
       "goals        146028.0      0.338058       0.906481  0.000000e+00     0.000000   \n",
       "yellowCards  146028.0      0.385364       0.795333  0.000000e+00     0.000000   \n",
       "yellowReds   146028.0      0.011381       0.107931  0.000000e+00     0.000000   \n",
       "redCards     146028.0      0.012559       0.112889  0.000000e+00     0.000000   \n",
       "rater1       124621.0      0.264255       0.295382  0.000000e+00     0.000000   \n",
       "rater2       124621.0      0.302862       0.293020  0.000000e+00     0.000000   \n",
       "refNum       146028.0   1534.827444     918.736625  1.000000e+00   641.000000   \n",
       "refCountry   146028.0     29.642842      27.496189  1.000000e+00     7.000000   \n",
       "meanIAT      145865.0      0.346276       0.032246 -4.725423e-02     0.334684   \n",
       "nIAT         145865.0  19697.411216  127126.197143  2.000000e+00  1785.000000   \n",
       "seIAT        145865.0      0.000631       0.004736  2.235373e-07     0.000055   \n",
       "meanExp      145865.0      0.452026       0.217469 -1.375000e+00     0.336101   \n",
       "nExp         145865.0  20440.233860  130615.745103  2.000000e+00  1897.000000   \n",
       "seExp        145865.0      0.002994       0.019723  1.043334e-06     0.000225   \n",
       "\n",
       "                     50%          75%           max  \n",
       "height        182.000000   187.000000  2.030000e+02  \n",
       "weight         76.000000    81.000000  1.000000e+02  \n",
       "games           2.000000     3.000000  4.700000e+01  \n",
       "victories       1.000000     2.000000  2.900000e+01  \n",
       "ties            0.000000     1.000000  1.400000e+01  \n",
       "defeats         1.000000     1.000000  1.800000e+01  \n",
       "goals           0.000000     0.000000  2.300000e+01  \n",
       "yellowCards     0.000000     1.000000  1.400000e+01  \n",
       "yellowReds      0.000000     0.000000  3.000000e+00  \n",
       "redCards        0.000000     0.000000  2.000000e+00  \n",
       "rater1          0.250000     0.250000  1.000000e+00  \n",
       "rater2          0.250000     0.500000  1.000000e+00  \n",
       "refNum       1604.000000  2345.000000  3.147000e+03  \n",
       "refCountry     21.000000    44.000000  1.610000e+02  \n",
       "meanIAT         0.336628     0.369894  5.737933e-01  \n",
       "nIAT         2882.000000  7749.000000  1.975803e+06  \n",
       "seIAT           0.000151     0.000229  2.862871e-01  \n",
       "meanExp         0.356446     0.588297  1.800000e+00  \n",
       "nExp         3011.000000  7974.000000  2.029548e+06  \n",
       "seExp           0.000586     0.001002  1.060660e+00  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "playerShort       object\n",
       "player            object\n",
       "club              object\n",
       "leagueCountry     object\n",
       "birthday          object\n",
       "height           float64\n",
       "weight           float64\n",
       "position          object\n",
       "games              int64\n",
       "victories          int64\n",
       "ties               int64\n",
       "defeats            int64\n",
       "goals              int64\n",
       "yellowCards        int64\n",
       "yellowReds         int64\n",
       "redCards           int64\n",
       "photoID           object\n",
       "rater1           float64\n",
       "rater2           float64\n",
       "refNum             int64\n",
       "refCountry         int64\n",
       "Alpha_3           object\n",
       "meanIAT          float64\n",
       "nIAT             float64\n",
       "seIAT            float64\n",
       "meanExp          float64\n",
       "nExp             float64\n",
       "seExp            float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['playerShort',\n",
       " 'player',\n",
       " 'club',\n",
       " 'leagueCountry',\n",
       " 'birthday',\n",
       " 'height',\n",
       " 'weight',\n",
       " 'position',\n",
       " 'games',\n",
       " 'victories',\n",
       " 'ties',\n",
       " 'defeats',\n",
       " 'goals',\n",
       " 'yellowCards',\n",
       " 'yellowReds',\n",
       " 'redCards',\n",
       " 'photoID',\n",
       " 'rater1',\n",
       " 'rater2',\n",
       " 'refNum',\n",
       " 'refCountry',\n",
       " 'Alpha_3',\n",
       " 'meanIAT',\n",
       " 'nIAT',\n",
       " 'seIAT',\n",
       " 'meanExp',\n",
       " 'nExp',\n",
       " 'seExp']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_columns = df.columns.tolist()\n",
    "all_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge\n",
    "\n",
    "Before looking below, try to answer some high level questions about the dataset. \n",
    "\n",
    "\n",
    "How do we operationalize the question of referees giving more red cards to dark skinned players?\n",
    "* Counterfactual: if the player were lighter, a ref is more likely to have given a yellow or no card **for the same offense under the same conditions**\n",
    "* Regression: accounting for confounding, darker players have positive coefficient on regression against proportion red/total card\n",
    "\n",
    "Potential issues\n",
    "* How to combine rater1 and rater2? Average them? What if they disagree? Throw it out?\n",
    "* Is data imbalanced, i.e. red cards are very rare?\n",
    "* Is data biased, i.e. players have different amounts of play time? Is this a summary of their whole career?\n",
    "* How do I know I've accounted for all forms of confounding?\n",
    "\n",
    "**First, is there systematic discrimination across all refs?**\n",
    "\n",
    "Exploration/hypotheses:\n",
    "* Distribution of games played\n",
    "* red cards vs games played\n",
    "* Reds per game played vs total cards per game played by skin color\n",
    "* Distribution of # red, # yellow, total cards, and fraction red per game played for all players by avg skin color\n",
    "* How many refs did players encounter?\n",
    "* Do some clubs play more aggresively and get carded more? Or are more reserved and get less?\n",
    "* Does carding vary by leagueCountry?\n",
    "* Do high scorers get more slack (fewer cards) for the same position?\n",
    "* Are there some referees that give more red/yellow cards than others?\n",
    "* how consistent are raters? Check with Cohen's kappa.\n",
    "* how do red cards vary by position? e.g. defenders get more?\n",
    "* Do players with more games get more cards, and is there difference across skin color?\n",
    "* indication of bias depending on refCountry?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understand how the data's organized\n",
    "\n",
    "The dataset is a single csv where it aggregated every interaction between referee and player into a single row. In other words: Referee A refereed Player B in, say, 10 games, and gave 2 redcards during those 10 games. Then there would be a unique row in the dataset that said: \n",
    "\n",
    "    Referee A, Player B, 2 redcards, ... \n",
    "\n",
    "This has several implications that make this first step to understanding and dealing with this data a bit tricky. First, is that the information about Player B is repeated each time -- meaning if we did a simple average of some metric of we would likely get a misleading result. \n",
    "\n",
    "For example, asking \"what is the average `weight` of the players?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.93593798236887"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['height'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.93593798236887"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['height'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181.74372848007872"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(df.groupby('playerShort').height.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tidy Data\n",
    "\n",
    "Hadley Wickham's concept of a **tidy dataset** summarized as:\n",
    "\n",
    ">  - Each variable forms a column\n",
    ">  - Each observation forms a row\n",
    ">  - Each type of observational unit forms a table\n",
    "\n",
    "A longer paper describing this can be found in this [pdf](https://www.jstatsoft.org/article/view/v059i10/v59i10.pdf).\n",
    "\n",
    "Having datasets in this form allows for much simpler analyses. So the first step is to try and clean up the dataset into a tidy dataset. \n",
    "\n",
    "The first step that I am going to take is to break up the dataset into the different observational units. By that I'm going to have separate tables (or dataframes) for: \n",
    "\n",
    " - players\n",
    " - clubs\n",
    " - referees\n",
    " - countries\n",
    " - dyads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Players Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data1</th>\n",
       "      <th>data2</th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.684572</td>\n",
       "      <td>-1.514165</td>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.393477</td>\n",
       "      <td>0.230970</td>\n",
       "      <td>a</td>\n",
       "      <td>two</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.033184</td>\n",
       "      <td>0.072844</td>\n",
       "      <td>b</td>\n",
       "      <td>one</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.387763</td>\n",
       "      <td>0.771857</td>\n",
       "      <td>b</td>\n",
       "      <td>two</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.010037</td>\n",
       "      <td>0.676437</td>\n",
       "      <td>a</td>\n",
       "      <td>one</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      data1     data2 key1 key2\n",
       "0 -0.684572 -1.514165    a  one\n",
       "1  0.393477  0.230970    a  two\n",
       "2 -1.033184  0.072844    b  one\n",
       "3  0.387763  0.771857    b  two\n",
       "4  0.010037  0.676437    a  one"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame({'key1':['a', 'a', 'b', 'b', 'a'],\n",
    "     'key2':['one', 'two', 'one', 'two', 'one'],\n",
    "     'data1':np.random.randn(5),\n",
    "     'data2':np.random.randn(5)})\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key1\n",
       "a   -0.093686\n",
       "b   -0.322711\n",
       "Name: data1, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grouped = df2['data1'].groupby(df['key1'])\n",
    "grouped.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "player_index = 'playerShort'\n",
    "player_cols = [#'player', # drop player name, we have unique identifier\n",
    "               'birthday',\n",
    "               'height',\n",
    "               'weight',\n",
    "               'position',\n",
    "               'photoID',\n",
    "               'rater1',\n",
    "               'rater2',\n",
    "              ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Count the unique variables (if we got different weight values, \n",
    "# for example, then we should get more than one unique value in this groupby)\n",
    "all_cols_unique_players = df.groupby('playerShort').agg({col:'nunique' for col in player_cols})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>position</th>\n",
       "      <th>rater2</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-hunt</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-lennon</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-ramsey</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abdelhamid-el-kaoutari</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        position  rater2  birthday  height  weight  photoID  \\\n",
       "playerShort                                                                   \n",
       "aaron-hughes                   1       1         1       1       1        1   \n",
       "aaron-hunt                     1       1         1       1       1        1   \n",
       "aaron-lennon                   1       1         1       1       1        1   \n",
       "aaron-ramsey                   1       1         1       1       1        1   \n",
       "abdelhamid-el-kaoutari         1       1         1       1       1        1   \n",
       "\n",
       "                        rater1  \n",
       "playerShort                     \n",
       "aaron-hughes                 1  \n",
       "aaron-hunt                   1  \n",
       "aaron-lennon                 1  \n",
       "aaron-ramsey                 1  \n",
       "abdelhamid-el-kaoutari       1  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_cols_unique_players.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>position</th>\n",
       "      <th>rater2</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [position, rater2, birthday, height, weight, photoID, rater1]\n",
       "Index: []"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If all values are the same per player then this should be empty (and it is!)\n",
    "all_cols_unique_players[all_cols_unique_players > 1].dropna().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# A slightly more elegant way to test the uniqueness\n",
    "all_cols_unique_players[all_cols_unique_players > 1].dropna().shape[0] == 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_subgroup(dataframe, g_index, g_columns):\n",
    "    \n",
    "    \"\"\"Helper function that creates a sub-table from the columns and runs a quick uniqueness test.\"\"\"\n",
    "    g = dataframe.groupby(g_index).agg({col:'nunique' for col in g_columns})\n",
    "    if g[g > 1].dropna().shape[0] != 0:\n",
    "        print(\"Warning: you probably assumed this had all unique values but it doesn't.\")\n",
    "    return dataframe.groupby(g_index).agg({col:'max' for col in g_columns})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>position</th>\n",
       "      <th>rater2</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>Center Back</td>\n",
       "      <td>0.00</td>\n",
       "      <td>08.11.1979</td>\n",
       "      <td>182.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>3868.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-hunt</th>\n",
       "      <td>Attacking Midfielder</td>\n",
       "      <td>0.25</td>\n",
       "      <td>04.09.1986</td>\n",
       "      <td>183.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>20136.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-lennon</th>\n",
       "      <td>Right Midfielder</td>\n",
       "      <td>0.25</td>\n",
       "      <td>16.04.1987</td>\n",
       "      <td>165.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>13515.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-ramsey</th>\n",
       "      <td>Center Midfielder</td>\n",
       "      <td>0.00</td>\n",
       "      <td>26.12.1990</td>\n",
       "      <td>178.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>94953.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abdelhamid-el-kaoutari</th>\n",
       "      <td>Center Back</td>\n",
       "      <td>0.25</td>\n",
       "      <td>17.03.1990</td>\n",
       "      <td>180.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>124913.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    position  rater2    birthday  height  \\\n",
       "playerShort                                                                \n",
       "aaron-hughes                     Center Back    0.00  08.11.1979   182.0   \n",
       "aaron-hunt              Attacking Midfielder    0.25  04.09.1986   183.0   \n",
       "aaron-lennon                Right Midfielder    0.25  16.04.1987   165.0   \n",
       "aaron-ramsey               Center Midfielder    0.00  26.12.1990   178.0   \n",
       "abdelhamid-el-kaoutari           Center Back    0.25  17.03.1990   180.0   \n",
       "\n",
       "                        weight     photoID  rater1  \n",
       "playerShort                                         \n",
       "aaron-hughes              71.0    3868.jpg    0.25  \n",
       "aaron-hunt                73.0   20136.jpg    0.00  \n",
       "aaron-lennon              63.0   13515.jpg    0.25  \n",
       "aaron-ramsey              76.0   94953.jpg    0.00  \n",
       "abdelhamid-el-kaoutari    73.0  124913.jpg    0.25  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "players = get_subgroup(df, player_index, player_cols)\n",
    "players.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def save_subgroup(dataframe, g_index, subgroup_name, prefix='raw_'):\n",
    "    save_subgroup_filename = \"\".join([prefix, subgroup_name, \".csv.gz\"])\n",
    "    dataframe.to_csv(save_subgroup_filename, compression='gzip', encoding='UTF-8')\n",
    "    test_df = pd.read_csv(save_subgroup_filename, compression='gzip', index_col=g_index, encoding='UTF-8')\n",
    "    # Test that we recover what we send in\n",
    "    if dataframe.equals(test_df):\n",
    "        print(\"Test-passed: we recover the equivalent subgroup dataframe.\")\n",
    "    else:\n",
    "        print(\"Warning -- equivalence test!!! Double-check.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>position</th>\n",
       "      <th>rater2</th>\n",
       "      <th>birthday</th>\n",
       "      <th>height</th>\n",
       "      <th>weight</th>\n",
       "      <th>photoID</th>\n",
       "      <th>rater1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaron-hughes</th>\n",
       "      <td>Center Back</td>\n",
       "      <td>0.00</td>\n",
       "      <td>08.11.1979</td>\n",
       "      <td>182.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>3868.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-hunt</th>\n",
       "      <td>Attacking Midfielder</td>\n",
       "      <td>0.25</td>\n",
       "      <td>04.09.1986</td>\n",
       "      <td>183.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>20136.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-lennon</th>\n",
       "      <td>Right Midfielder</td>\n",
       "      <td>0.25</td>\n",
       "      <td>16.04.1987</td>\n",
       "      <td>165.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>13515.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaron-ramsey</th>\n",
       "      <td>Center Midfielder</td>\n",
       "      <td>0.00</td>\n",
       "      <td>26.12.1990</td>\n",
       "      <td>178.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>94953.jpg</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abdelhamid-el-kaoutari</th>\n",
       "      <td>Center Back</td>\n",
       "      <td>0.25</td>\n",
       "      <td>17.03.1990</td>\n",
       "      <td>180.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>124913.jpg</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    position  rater2    birthday  height  \\\n",
       "playerShort                                                                \n",
       "aaron-hughes                     Center Back    0.00  08.11.1979   182.0   \n",
       "aaron-hunt              Attacking Midfielder    0.25  04.09.1986   183.0   \n",
       "aaron-lennon                Right Midfielder    0.25  16.04.1987   165.0   \n",
       "aaron-ramsey               Center Midfielder    0.00  26.12.1990   178.0   \n",
       "abdelhamid-el-kaoutari           Center Back    0.25  17.03.1990   180.0   \n",
       "\n",
       "                        weight     photoID  rater1  \n",
       "playerShort                                         \n",
       "aaron-hughes              71.0    3868.jpg    0.25  \n",
       "aaron-hunt                73.0   20136.jpg    0.00  \n",
       "aaron-lennon              63.0   13515.jpg    0.25  \n",
       "aaron-ramsey              76.0   94953.jpg    0.00  \n",
       "abdelhamid-el-kaoutari    73.0  124913.jpg    0.25  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "players = get_subgroup(df, player_index, player_cols)\n",
    "players.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(players, player_index, \"players\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Clubs Table\n",
    "\n",
    "Create the clubs table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>leagueCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>club</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1. FC Nürnberg</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1. FSV Mainz 05</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1899 Hoffenheim</th>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AC Ajaccio</th>\n",
       "      <td>France</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AFC Bournemouth</th>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                leagueCountry\n",
       "club                         \n",
       "1. FC Nürnberg        Germany\n",
       "1. FSV Mainz 05       Germany\n",
       "1899 Hoffenheim       Germany\n",
       "AC Ajaccio             France\n",
       "AFC Bournemouth       England"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "club_index = 'club'\n",
    "club_cols = ['leagueCountry']\n",
    "clubs = get_subgroup(df, club_index, club_cols)\n",
    "clubs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "England    48\n",
       "Spain      27\n",
       "France     22\n",
       "Germany    21\n",
       "Name: leagueCountry, dtype: int64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clubs['leagueCountry'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(clubs, club_index, \"clubs\", )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Referees Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>refCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        refCountry\n",
       "refNum            \n",
       "1                1\n",
       "2                2\n",
       "3                3\n",
       "4                4\n",
       "5                5"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referee_index = 'refNum'\n",
    "referee_cols = ['refCountry']\n",
    "referees = get_subgroup(df, referee_index, referee_cols)\n",
    "referees.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "161"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.refCountry.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>refCountry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3143</th>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3144</th>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3145</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3146</th>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3147</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        refCountry\n",
       "refNum            \n",
       "3143            51\n",
       "3144            55\n",
       "3145            21\n",
       "3146            51\n",
       "3147            21"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3147, 1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "referees.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(referees, referee_index, \"referees\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Tidy Countries Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Alpha_3</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>nExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GRC</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>712.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.002696</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>750.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ZMB</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>40.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.061504</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.001002</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>1897.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LUX</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>127.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.013752</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>130.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>TUN</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.111757</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Alpha_3   meanIAT    nIAT   meanExp     seExp     seIAT    nExp\n",
       "refCountry                                                                \n",
       "1              GRC  0.326391   712.0  0.396000  0.002696  0.000564   750.0\n",
       "2              ZMB  0.203375    40.0 -0.204082  0.061504  0.010875    49.0\n",
       "3              ESP  0.369894  1785.0  0.588297  0.001002  0.000229  1897.0\n",
       "4              LUX  0.325185   127.0  0.538462  0.013752  0.003297   130.0\n",
       "5              TUN  0.167132    19.0 -0.789474  0.111757  0.027327    19.0"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_index = 'refCountry'\n",
    "country_cols = ['Alpha_3', # rename this name of country\n",
    "                'meanIAT',\n",
    "                'nIAT',\n",
    "                'seIAT',\n",
    "                'meanExp',\n",
    "                'nExp',\n",
    "                'seExp',\n",
    "               ]\n",
    "countries = get_subgroup(df, country_index, country_cols)\n",
    "countries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>countryName</th>\n",
       "      <th>meanIAT</th>\n",
       "      <th>nIAT</th>\n",
       "      <th>meanExp</th>\n",
       "      <th>seExp</th>\n",
       "      <th>seIAT</th>\n",
       "      <th>nExp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refCountry</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GRC</td>\n",
       "      <td>0.326391</td>\n",
       "      <td>712.0</td>\n",
       "      <td>0.396000</td>\n",
       "      <td>0.002696</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>750.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ZMB</td>\n",
       "      <td>0.203375</td>\n",
       "      <td>40.0</td>\n",
       "      <td>-0.204082</td>\n",
       "      <td>0.061504</td>\n",
       "      <td>0.010875</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ESP</td>\n",
       "      <td>0.369894</td>\n",
       "      <td>1785.0</td>\n",
       "      <td>0.588297</td>\n",
       "      <td>0.001002</td>\n",
       "      <td>0.000229</td>\n",
       "      <td>1897.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LUX</td>\n",
       "      <td>0.325185</td>\n",
       "      <td>127.0</td>\n",
       "      <td>0.538462</td>\n",
       "      <td>0.013752</td>\n",
       "      <td>0.003297</td>\n",
       "      <td>130.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>TUN</td>\n",
       "      <td>0.167132</td>\n",
       "      <td>19.0</td>\n",
       "      <td>-0.789474</td>\n",
       "      <td>0.111757</td>\n",
       "      <td>0.027327</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           countryName   meanIAT    nIAT   meanExp     seExp     seIAT    nExp\n",
       "refCountry                                                                    \n",
       "1                  GRC  0.326391   712.0  0.396000  0.002696  0.000564   750.0\n",
       "2                  ZMB  0.203375    40.0 -0.204082  0.061504  0.010875    49.0\n",
       "3                  ESP  0.369894  1785.0  0.588297  0.001002  0.000229  1897.0\n",
       "4                  LUX  0.325185   127.0  0.538462  0.013752  0.003297   130.0\n",
       "5                  TUN  0.167132    19.0 -0.789474  0.111757  0.027327    19.0"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rename_columns = {'Alpha_3':'countryName', }\n",
    "countries = countries.rename(columns=rename_columns)\n",
    "countries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(161, 7)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "countries.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning -- equivalence test!!! Double-check.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(countries, country_index, \"countries\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create separate (not yet Tidy) Dyads Table\n",
    "\n",
    "This is one of the more complex tables to reason about -- so we'll save it for a bit later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dyad_index = ['refNum', 'playerShort']\n",
    "dyad_cols = ['games',\n",
    "             'victories',\n",
    "             'ties',\n",
    "             'defeats',\n",
    "             'goals',\n",
    "             'yellowCards',\n",
    "             'yellowReds',\n",
    "             'redCards',\n",
    "            ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dyads = get_subgroup(df, g_index=dyad_index, g_columns=dyad_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>redCards</th>\n",
       "      <th>victories</th>\n",
       "      <th>defeats</th>\n",
       "      <th>goals</th>\n",
       "      <th>games</th>\n",
       "      <th>yellowCards</th>\n",
       "      <th>ties</th>\n",
       "      <th>yellowReds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>lucas-wilchez</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>john-utaka</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">3</th>\n",
       "      <th>abdon-prats</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pablo-mari</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ruben-pena</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      redCards  victories  defeats  goals  games  yellowCards  \\\n",
       "refNum playerShort                                                              \n",
       "1      lucas-wilchez         0          0        1      0      1            0   \n",
       "2      john-utaka            0          0        1      0      1            1   \n",
       "3      abdon-prats           0          0        0      0      1            1   \n",
       "       pablo-mari            0          1        0      0      1            0   \n",
       "       ruben-pena            0          1        0      0      1            0   \n",
       "\n",
       "                      ties  yellowReds  \n",
       "refNum playerShort                      \n",
       "1      lucas-wilchez     0           0  \n",
       "2      john-utaka        0           0  \n",
       "3      abdon-prats       1           0  \n",
       "       pablo-mari        0           0  \n",
       "       ruben-pena        0           0  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(146028, 8)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>yellowCards</th>\n",
       "      <th>yellowReds</th>\n",
       "      <th>victories</th>\n",
       "      <th>ties</th>\n",
       "      <th>games</th>\n",
       "      <th>defeats</th>\n",
       "      <th>goals</th>\n",
       "      <th>redCards</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>refNum</th>\n",
       "      <th>playerShort</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <th>bodipo</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <th>antonio-lopez_2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">432</th>\n",
       "      <th>javi-martinez</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jonas</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>487</th>\n",
       "      <th>phil-jagielka</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>586</th>\n",
       "      <th>cyril-jeunechamp</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>804</th>\n",
       "      <th>sergio-ramos</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <th>aly-cissokho</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1114</th>\n",
       "      <th>eugen-polanski</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1214</th>\n",
       "      <th>emmanuel-adebayor</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>23</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          yellowCards  yellowReds  victories  ties  games  \\\n",
       "refNum playerShort                                                          \n",
       "140    bodipo                       0           0          2     1      6   \n",
       "367    antonio-lopez_2              2           0          5     2      8   \n",
       "432    javi-martinez                2           0          4     3     14   \n",
       "       jonas                        0           0          1     4      9   \n",
       "487    phil-jagielka                0           0          2     1      7   \n",
       "586    cyril-jeunechamp             6           0          8     0     14   \n",
       "804    sergio-ramos                 6           1         12     1     18   \n",
       "985    aly-cissokho                 1           0          1     5      9   \n",
       "1114   eugen-polanski               0           0          4     0      8   \n",
       "1214   emmanuel-adebayor            4           1          9     7     23   \n",
       "\n",
       "                          defeats  goals  redCards  \n",
       "refNum playerShort                                  \n",
       "140    bodipo                   3      1         2  \n",
       "367    antonio-lopez_2          1      0         2  \n",
       "432    javi-martinez            7      2         2  \n",
       "       jonas                    4      1         2  \n",
       "487    phil-jagielka            4      1         2  \n",
       "586    cyril-jeunechamp         6      0         2  \n",
       "804    sergio-ramos             5      4         2  \n",
       "985    aly-cissokho             3      1         2  \n",
       "1114   eugen-polanski           4      0         2  \n",
       "1214   emmanuel-adebayor        7     10         2  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads[dyads.redCards > 1].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test-passed: we recover the equivalent subgroup dataframe.\n"
     ]
    }
   ],
   "source": [
    "save_subgroup(dyads, dyad_index, \"dyads\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dyads.redCards.max()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "318px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
